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Abstract 

In this paper we complete the global qualitative analysis of a quartic ecological 
model. In particular, studying global bifurcations of singular points and limit cycles, 
we prove that the corresponding dynamical system has at most two limit cycles. 
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1 Introduction 

The paper is based on the applications of Bifurcation Theory and can be used 
for modeling problems, where system parameters play a certain role in various 
bifurcations. In this paper we consider a particular (quartic) family of planar 
vector fields, which models the dynamics of the populations of predators and 
their prey in a given ecological system and which is a variation on the classical 
Lotka-Volterra system. For the latter system the change of the prey density 
per unit of time per predator called the response function is proportional to 
the prey density. This means that there is no saturation of the predator when 
the amount of available prey is large. However, it is more realistic to consider 
a nonlinear and bounded response function, and in fact different response 
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functions have been used in the literature to model the predator response, see 

m m, nu-nn, m- 



For instance, Zhu et al. [16] have studied recently the following predator-prey 
model: 

x = x(a — Xx) — yP(x) (prey), 

(1.1) 

y = —5y + yQ{x) (predator). 

The variables x > and y > denote the density of the prey and predator 
populations respectively, while P(x) is a non-monotonic response function 
given by 

P{x)= TZp7i XT ' ( L2 ) 
ax z + px + 1 

where a, m are positive and where j3 > —l^fa. Observe that in the absence 
of predators, the number of prey increases according to a logistic growth law. 
The coefficient a represents the intrinsic growth rate of the prey, while A > 
is the rate of competition or resource limitation of prey. The natural death 
rate of the predator is given by 5 > 0. In Gause's model the function Q(x) is 
given by Q(x) = cP(x), where c > is the rate of conversion between prey 
and predator. For further discussion on the biological relevance of the model 
see [3] [5J. 

In this paper we investigate the following family 



i = X { 1 - Xx - axi + Px + l ) (prey) ' 



1.3) 



where a >0, 5 > 0, A>0, fi >0 and (3 > — 2 v / a are parameters. We note that 
(1.3) is obtained from (1.1) by adding the term — \iy 2 to the second equation 
and after scaling x and y, as well as the parameters and the time t. In this 
way we take into account competition between predators for resources other 
than prey. The non-negative coefficient fi is the rate of competition amongst 
predators. For examples of populations that use the group defense strategy 
see [6J. 

System (1.3) can be written in the form 

x = x((l — Ax) (ax 2 + /3x + 1) — y) = P, 

(1.4) 

y = —y((8 + jj,y){ax 2 + f3x + 1) — x) = Q. 

This quartic ecological model was studied earlier, for instance, in [3j-[6j. How- 
ever, the qualitative analysis was incomplete, since the global bifurcations of 
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limit cycles could not be studied properly by means of the methods and tech- 
niques which were used earlier in the qualitative theory of dynamical systems. 

Together with (1.4), we will also consider an auxiliary system (see [T], [T5] ) 



applying to these systems new bifurcation methods and geometric approaches 
developed in [7J [E] and completing the qualitative analysis of (1.4). 



2 Preliminaries 

In this paper geometric aspects of Bifurcation Theory are used and developed. 
First of all, the two-isocline method which was developed by Erugin is used, 
see [7J. An isocline portrait is the most natural construction for a polyno- 
mial equation. It is sufficient to have only two isoclines (of zero and infinity) 
to obtain principal information on the original polynomial system, because 
these two isoclines are right-hand sides of the system. Geometric properties of 
isoclines (conies, cubics, quartics, etc.) are well-known, and all isocline por- 
traits can be easily constructed. By means of them, all topologically different 
qualitative pictures of integral curves to within a number of limit cycles and 
distinguishing center and focus can be obtained. Thus, it is possible to carry 
out a rough topological classification of the phase portraits for the polyno- 
mial dynamical systems. It is the first application of Erugin's method. After 
studying contact and rotation properties of the isoclines, the simplest (canon- 
ical) systems containing limit cycles can be also constructed. Two groups of 
parameters can be distinguished in such systems: static and dynamic. Static 
parameters determine the behavior of phase trajectories in principle, since 
they control the number, position, and character of singular points in a finite 
part of the plane (finite singularities). The parameters from the first group 
determine also a possible behavior of separatrices and singular points at infin- 
ity (infinite singularities) under variation of the parameters from the second 
group. The dynamic parameters are field rotation parameters, see [T], [7J, [To] . 
They do not change the number, position and index of the finite singularities, 
but only involve the vector field in a directional rotation. The rotation pa- 
rameters allow to control the infinite singularities, the behavior of limit cycles 
and separatrices. The cyclicity of singular points and separatrix cycles, the 
behavior of semi-stable and other multiple limit cycles are controlled by these 
parameters as well. Therefore, by means of the rotation parameters, it is pos- 
sible to control all limit cycle bifurcations and to solve the most complicated 
problems of the qualitative theory of dynamical systems. 

In [7J, [H] some complete results on quadratic systems have been presented. 



x = P - jQ 



y = Q + 1 P, 
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In particular, it has been proved that for quadratic systems four is really the 
maximum number of limit cycles and (3 : 1), i.e., three limit cycles around 
one focus and the only limit cycle around another focus, is their only possible 
distribution (this is a solution of Hilbert's Sixteenth Problem in the quadratic 
case of polynomial dynamical systems). In [10] some preliminary results on 
generalizing new ideas and methods of [7] to cubic dynamical systems have 
already been established. In particular, a canonical cubic system of Kukles 
type has been constructed and the global qualitative analysis of its special 
case corresponding to a generalized Lienard equation has been carried out. It 
has been proved also that the foci of such a Lienard system can be at most of 
second order and that such system can have at most three limit cycles on the 
whole phase plane. Moreover, unlike all previous works on the Kukles-type 
systems, global bifurcations of limit and separatrix cycles using arbitrary (in- 
cluding as large as possible) field rotation parameters of the canonical system 
have been studied in [TU]. As a result, the classification of all possible types 
of separatrix cycles for the generalized Lienard system has been obtained and 
all possible distributions of its limit cycles have been found. In [9] a solution 
of Smale's Thirteenth Problem proving that the Lienard system with a poly- 
nomial of degree 2k + 1 can have at most k limit cycles has been presented. 
All of these methods and results can be applied to quartic dynamical systems 
as well. In this paper, using [7] [10], we will complete the global qualitative 
analysis of quartic ecological model (1.4). In particular, studying global bifur- 
cations of singular points and limit cycles, we will prove that the corresponding 
dynamical system has at most two limit cycles. 



3 Singular Points 

The study of singular point of system (1.4) will use two index theorems by 
H. Poincare, see [T]. But first let us define the Poincare index [Tj. 

Definition 3.1. Let S be a simple closed curve in the phase plane not passing 
through a singular point of the system 



where P(x, y) and Q(x, y) are continuous functions (for example, polynomials), 
and M be some point on S. If the point M goes around the curve S in positive 
direction (counterclockwise) one time, then the vector coinciding with the 
direction of a tangent to the trajectory passing through the point M is rotated 
through the angle 2irj (j = 0, ±1, ±2, . . .). The integer j is called the Poincare 
index of the closed curve S relative to the vector field of system (3.1) and has 



x = P(x,y), y = Q(x,y), 
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the expression 

. _ 1 rPdQ-QdP 

3 ~ 2nf P 2 + Q 2 ' 

s 

According to this definition, the index of a node or a focus, or a center is equal 
to +1 and the index of a saddle is —1. 

Theorem 3.1 (First Poincare Index Theorem). If N, Nf, N c , and C are 

respectively the number of nodes, foci, centers, and saddles in a finite part of 
the phase plane and N' and C are the number of nodes and saddles at infinity, 
then it is valid the formula 

N + N f + N c + N' = C + C' + l. 

Theorem 3.2 (Second Poincare Index Theorem). If all singular points 
are simple, then along an isocline without multiple points lying in a Poincare 
hemisphere which is obtained by a stereographic projection of the phase plane, 
the singular points are distributed so that a saddle is followed by a node or a 
focus, or a center and vice versa. If two points are separated by the equator 
of the Poincare sphere, then a saddle will be followed by a saddle again and a 
node or a focus, or a center will be followed by a node or a focus, or a center. 

We will use also the following theorem by A. N. Berlinskii, see [Tj. 

Theorem 3.3 (Berlinskii Theorem). If a quadratic system (3.1) has four 
singular points in a finite part of the phase plane, then only one of the fol- 
lowing cases is possible: 1) these points are vertices of a convex quadrangular, 
where two opposite vertices are saddles (antisaddles) and two others are anti- 
saddles (saddles); 2) three singular points are vertices of a triangle containing 
the fourth point inside, and if this point is a saddle (antisaddle) , then the oth- 
ers are antisaddles (saddles), where antisaddles are singularities that are not 
a saddle. 

Consider system (1.4) which has two invariant straight lines: x = and y = 0. 
Its finite singularities are determined by the algebraic system 

x((l — Xx)(ax 2 + fix + 1) — y) = 0, 

(3.2) 

y((5 + fiy)(ax 2 + fix + 1) - x) = 0. 

From (3.2), we have got: two singular points (0, 0) and (0, —5/fi), at most two 
points defined by the condition 

ax 2 + fix + 1 = 0, y = 0, (3.3) 
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and at most four singularities defined by the system 



y = (1- Xx)(ax 2 + fix + 1), 

(3.4) 

y (5 + fiy) — x (1 — Xx) = 0, 
among which we always have the point (1/A, 0). 

To investigate the character and distribution of the singular points in the phase 
plane, we will use the method developed in [7] [10]. The sense of this method 
is to obtain the simplest (well-known) system by vanishing some parameters 
(usually field rotation parameters) of the original system and then to input 
these parameters successively one by one studying the dynamics of the singular 
points (both finite and infinite) in the phase plane. 

Let the parameters a, (3 vanish and consider first the quadratic system 

x = x(l — Xx — y), 

(3.5) 

y = -y(S + fiy-x). 

System (3.5) has four finite singularities, if 5 ^ 1/A. Studying isocline portraits 
of the equation corresponding to system (3.5) and applying theorems 3.1-3.3, 
we can see that for the case, when 5 > 1/A, system (3.5) has two saddles: 
(0, 0) and a point of intersection of two straight line-isoclines: 

1 — Xx — y = 0, 5 + fiy — x = 0, (3.6) 

- and two nodes: (0, —5/fi) and (1/A,0). For the case, when 6 < 1/A, system 

(3.5) has two saddles: (0,0) and (1/A, 0), — and two nodes: (0,—5/fi) and 

(3.6) . If 5 = 1/A, it has three singularities: a saddle (0,0), a node (0, —5/ji), 
and a saddle-node (1/A, 0). Since we consider the first coordinate quadrant 
with respect to the variables x and y, we will be interested basically in the case 
of S < 1/A, when the singular point defined by (3.6) is in the first quadrant. 

To study singular points at infinity, consider the corresponding differential 
equation 

dy = y(6 + ny - x) 
dx x(l — Ax — y) 

Dividing the numerator and denominator of the right-hand side of (3.7) by x 2 
(x 7^ 0) and denoting y/x by u, we will get the algebraic equation 

(1 — /j)u 2 + (1 + X)u = 0, where u = y/x, (3.8) 

for all infinite singularities of (3.7) except when x = (the "ends" of the 
y-axis), see P, [7]. For this special case we can divide the numerator and 
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denominator of the right-hand side of (3.7) by y 2 (y ^ 0) denoting x/y by v 
and consider the algebraic equation 

(1 + X)v 2 + (1 - n)v = 0, where v = x/y. (3.9) 

The equations (3.8) and (3.9) give three singular points at infinity for (3.7): 
two nodes on the "ends" of the x and y axes and a saddle in the direction of 
u=(A + 1)/(ax-1). 

Fix the parameters 5, X, \x and take (3 < (this case will be also more inte- 
resting to us). After inputting the parameter (3, we will have a cubic system: 



x — x((l — Xx){(3x + 1) — y), 

(3.10) 

y = -y(($ + v>y)(Px + 1) - x). 

For 5 < 1/A and /3 < 0, system (3.10) has five finite singularities: two sad- 
dles - - (0,0) and (1/A, 0), two nodes - - (0,—5/fi) and (—1/(3,0), and an 
antisaddle (a node, a focus, or a center) defined as a point of intersection of 
two isoclines: 

(1 - Xx)((3x + 1) - y = 0, 

(3.11) 

(6 + fj,y)(J3x + l)-x = 0. 
For singular points at infinity, consider the corresponding differential equation 

dy _ y((5 + fxy)(px + 1) - x) 

dx x((l-Xx)((3x + l)-y) 1 ' ' 

and the algebraic equations 

/iw 2 — Am = 0, where u = y/x, (3.13) 

and 

Xv 3 — fiv 2 = 0, where v — x/y, (3-14) 

which give three infinite singularities: a node on the "ends" of the x-axis, 
a saddle-node on the "ends" of the y-axis, and a saddle in the direction of 
u = X/ ji. 

Fix the parameters (3, 5, X, « and take a > 0. Studying the bundle of cubic 
curves 

y = (1 - Xx)(ax 2 + px+ 1) (3.15) 

which intersect at the point (1/A, 0) and contact at the point (0, 1), we can 
see that system (1.4) obtained after inputting a has first six finite singular 



points: three saddles — (0,0), (1/A,0), and {{-(3+ J f3 2 - 4a )/(2a), 0), two 
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nodes — (0, —S/fi) and ((— /? — yj ' f3 2 — 4a )/(2a), 0), and an antisaddle defined 
as a point of intersection of isoclines (3.4). 



On increasing the parameter a, the points ((— /3 — \J ' (3 2 — 4a )/(2a), 0) and 

((—(3 + J ' (3 2 — 4a ) / (2a), 0) combine a saddle-node which then disappears. On 
further increasing a, the point (1/A, 0) becomes a triple saddle from which a 
saddle and a node (or a saddle-node) will appear. Thus, we will have three 
singular points in the first quadrant: a saddle S and two antisaddles — A x 
and A 2 which are defined as points of intersection of isoclines (3.4). Suppose 
that with respect to the x-axis they have the following sequence: Ai, S, A 2 . 

To study singular points of (1.4) at infinity, consider the corresponding diffe- 
rential equation 

dy_ _ y((S + ny)(ax 2 + f3x + 1) - x) 

dx~ x((l - \x)(ax 2 + px + 1) - y) ^' ' 

and the algebraic equations 

yU-u 2 — Am = 0, where u = y/x, (3-17) 

and 

Af 4 — /if 3 = 0, where v = x/y, (3.18) 

which give three infinite singularities: a simple node on the "ends" of the 
a;-axis, a triple node on the "ends" of the y-axis, and a simple saddle in the 
direction of u = A///. 

Note that all results on finite singularities of system (1.4) agree with the results 
of [3]-[6], [H], [16] . but where infinite singularities have not been investigated 
at all. Using the obtained information and applying the approach developed 
in [7] -[ID], we can study limit cycle bifurcations of system (1.4) now. This 
study will use also some results obtained in [3]-[6], [H], [IE]. In particular, 
the results on the cyclicity of singular points of (1.4) will be used. However, 
it is surely not enough to have only these results to prove the main theorem 
of this paper: on the maximum number of limit cycles of (1.4). 

4 Bifurcations of Limit Cycles 



Let us first formulate the Wintner-Perko termination principle [15] for the 
polynomial system 

x = f(x,v), (4.1/x) 
where x e R 2 ; /x G R n ; / G R 2 (/ is a polynomial vector function). 
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Theorem 4.1 (Wintrier- Perko termination principle). Any one-para- 
meter family of multiplicity-m limit cycles of relatively prime polynomial sys- 
tem (4.1u) can be extended in a unique way to a maximal one-parameter family 
of multiplicity-m limit cycles o/(4.1#t) which is either open or cyclic. 

If it is open, then it terminates either as the parameter or the limit cycles be- 
come unbounded; or, the family terminates either at a singular point of (4.1^), 
which is typically a fine focus of multiplicity m, or on a (compound) separatrix 
cycle of (4.1 ^x), which is also typically of multiplicity m. 

The proof of this principle for general polynomial system (4.1^) with a vector 
parameter fj, e R n parallels the proof of the planar termination principle for 

the system 

x — P(x,y, A), y = Q(x,y,X) (4.1 A ) 

with a single parameter A G R (see [7j , [I_5] ) , since there is no loss of generality 
in assuming that system (4.1^) is parameterized by a single parameter A; i. e., 
we can assume that there exists an analytic mapping £t(A) of R into R™ such 
that (4.1^) can be written as (4.1 «(a)) or even (4.1a) and then we can repeat 
everything, what had been done for system (4.1^) in [15] . In particular, if A is a 
field rotation parameter of (4.1^), the following Perko's theorem on monotonic 
families of limit cycles is valid (see [To]). 

Theorem 4.2. If L Q is a nonsingular multiple limit cycle of (4.1q), then Lq 
belongs to a one-parameter family of limit cycles of (4.1 a); furthermore: 

1) if the multiplicity of Lq is odd, then the family either expands or contracts 
monotonically as A increases through Ao; 

2) if the multiplicity of L is even, then L bifurcates into a stable and an 
unstable limit cycle as A varies from Xq in one sense and L disappears as A 
varies from Xq in the opposite sense; i. e., there is a fold bifurcation at Xq. 

Applying the definition of a field rotation parameter [TJ, [TJ, [15], i. e., a param- 
eter which rotates the field in one direction, to system (1.4), let us calculate 
the corresponding determinants for the parameters a and /3, respectively: 

A Q = PQ' a - QP' a = x 3 y(y(6 + py) - ar(l - Xx)), (4.2) 

Ap = PQ' p - QP' p = x 2 y(y(5 + fiy) - x(l - Ax)). (4.3) 

It follows from (4.2) and (4.3) that on increasing a or (3 the vector field of 
(1.4) in the first quadrant is rotated in positive direction (counterclockwise) 
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only on the outside of the ellipse 



y(S + nv) ~ x{l - Xx) = 0. (4.4) 

Therefore, to study limit cycle bifurcations of system (1.4), it makes sense 
together with (1.4) to consider also an auxiliary system (1.5) with a field 
rotation parameter 7 : 

A 7 = P 2 + Q 2 > 0. (4.5) 

Using system (1.5) and applying Perko's results, we will prove the following 
theorem. 

Theorem 4.3. System (1.4) has at most two limit cycles. 

Proof. First let us prove that system (1.4) can have at least two limit cycles. 
Begin with quadratic system (3.5). It is clear that such a system, with two 
invariant straight lines, cannot have limit cycles at all [TJ. Inputting a nega- 
tive parameter (3 into this system, the vector field of cubic system (3.10) will 
be rotated in negative direction (clockwise) at infinity, the structure and the 
character of stability of infinite singularities will be changed, and an unstable 
limit, Fx, will appear immediately from infinity in this case. This cycle will 
surround a stable antisaddle (a node or a focus) A\ which is in the first quad- 
rant of system (3.10). Inputting a positive parameter a into system (3.10), 
the vector field of quartic system (1.4) will be rotated in positive direction 
(counterclockwise) at infinity, the structure and the character of stability of 
infinite singularities will be changed again, and a stable limit, T 2 , surrounding 
I\ will appear immediately from infinity in this case. On further increasing 
the parameter a, the limit cycles T\ and T2 combine a semi-stable limit, Ti2, 
which then disappears in a "trajectory concentration" [1], [7J. 

As we saw above, on further increasing a, two other singular points, a saddle 
S and an antisaddle A 2 , will appear in the first quadrant in system (1.4). 
We can fix the parameter a, fixing simultaneously the positions of the finite 
singularities Ai, S, A 2 , and consider system (1.5) with a positive parameter 7 
which acts like a positive parameter a of system (1.4), but on the whole phase 
plane. 

So, consider system (1.5) with a positive parameter 7. On increasing this pa- 
rameter, the stable nodes A\ and A 2 becomes first stable foci, then they change 
the character of their stability, becoming unstable foci. At these Andronov- 
Hopf bifurcations [T], [7J, stable limit cycles will appear from the foci A\ and 
A 2 . On further increasing 7, the limit cycles will expand and will disappear 
in small separatrix loops of the saddle S. If these loops are formed simultane- 
ously, we will have a so-called eight-loop separatrix cycle. In this case, a big 
stable limit surrounding three singular points, A%, S, and A 2 , will appear from 
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the eight-loop separatrix cycle after its destruction, expanding to infinity on 
increasing 7. If a small loop is formed earlier, for example, around the point Ai 
{A 2 ), then, on increasing 7, a big loop formed by two lower (upper) adjoining 
separatrices of the saddle S and surrounding the points A\ and A 2 will appear. 
After its destruction, we will have simultaneously a big limit cycle surrounding 
three singular points, Ai, S, A 2 , and a small limit cycle surrounding the point 
A 2 (Ai). Thus, we have proved that system (1.4) can have at least two limit 
cycles, see also [3] [6], [TJ], [T6] . 

Let us prove now that this system has at most two limit cycles. The proof 
is carried out by contradiction applying Catastrophe Theory, see [TJ, [T5] . 
Consider system (1.5) with three parameters: a, (3, and 7 (the parameters 
5, A, and \i can be fixed, since they do not generate limit cycles). Suppose 
that (1.5) has three limit cycles surrounding the only point, A\, in the first 
quadrant. Then we get into some domain of the parameters a, /3, and 7 being 
restricted by definite conditions on three other parameters, 5, A, and fi. This 
domain is bounded by two fold bifurcation surfaces forming a cusp bifurcation 
surface of multiplicity-three limit cycles in the space of the parameters a, f3, 
and 7 H|, [15]. 

The corresponding maximal one-parameter family of multiplicity-three limit 
cycles cannot be cyclic, otherwise there will be at least one point corresponding 
to the limit cycle of multiplicity four (or even higher) in the parameter space. 
Extending the bifurcation curve of multiplicity-four limit cycles through this 
point and parameterizing the corresponding maximal one-parameter family of 
multiplicity-four limit cycles by the field rotation parameter, 7, according to 
Theorem 4.2, we will obtain two monotonic curves of multiplicity-three and 
one, respectively, which, by the Wintner-Perko termination principle (Theo- 
rem 4.1), terminate either at the point A\ or on a separatrix cycle surrounding 
this point. Since we know at least the cyclicity of the singular point which is 
equal to two (see [3] — [H] , [H], [IS]), we have got a contradiction with the termi- 
nation principle stating that the multiplicity of limit cycles cannot be higher 
than the multiplicity (cyclicity) of the singular point in which they terminate. 

If the maximal one-parameter family of multiplicity-three limit cycles is not 
cyclic, using the same principle (Theorem 4.1), this again contradicts the 
cyclicity of A\ (see [3] [6], [TJ], [16]) not admitting the multiplicity of limit 
cycles to be higher than two. This contradiction completes the proof in the 
case of one singular point in the first quadrant. 

Suppose that system (1.5) with three finite singularities, A lf S, and A 2 , has 
two small limit cycles around, for example, the point A\ (the case when limit 
cycles surround the point A 2 is considered in a similar way). Then we get into 
some domain in the space of the parameters a, f3, and 7 which is bounded by 
a fold bifurcation surface of multiplicity- two limit cycles [7J , [T5] . 
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The corresponding maximal one-parameter family of multiplicity-two limit 
cycles cannot be cyclic, otherwise there will be at least one point corresponding 
to the limit cycle of multiplicity three (or even higher) in the parameter space. 
Extending the bifurcation curve of multiplicity-three limit cycles through this 
point and parameterizing the corresponding maximal one-parameter family 
of multiplicity-three limit cycles by the field rotation parameter, 7, according 
to Theorem 4.2, we will obtain a monotonic curve which, by the Wintner- 
Perko termination principle (Theorem 4.1), terminates either at the point Ai 
or on some separatrix cycle surrounding this point. Since we know at least the 
cyclicity of the singular point which is equal to one in this case [3] — [0] , [T4] . 
|16j . we have got a contradiction with the termination principle (Theorem 4.1). 

If the maximal one-parameter family of multiplicity-two limit cycles is not 
cyclic, using the same principle (Theorem 4.1), this again contradicts the 
cyclicity of Ai (see [3] [6], [H], [16]) not admitting the multiplicity of limit 
cycles higher than one. Moreover, it also follows from the termination principle 
that either an ordinary (small) separatrix loop or a big loop, or an eight-loop 
cannot have the multiplicity (cyclicity) higher than one in this case. Therefore, 
according to the same principle, there are no more than one limit cycle in the 
exterior domain surrounding all three finite singularities, Ax, S, and A2. 

Thus, taking into account all other possibilities for limit cycle bifurcations 
(see [3] [6], [H], [16]), we conclude that system (1.4) cannot have either a 
multiplicity-three limit cycle or more than two limit cycles in any configura- 
tion. The theorem is proved. □ 
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